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Abstract 



Colleague and student ratings were gathered on a group of 477 irscructors 
and then compared to the instructor's research productivity and academic rank. 
Colleague and student ratings v;ere not found to be significantly related to the 
Instructor's research productivity • However, colleague rating was significantly 
related to academic rank indicating that the reputation of the instructor could 
be influencing colleague ratings. 



An Investigation of the Relationship Between 
Colleckgue Rating, Student Rating, Research Productivity, 
and Academic Rank in Rating Instructional Effectiveness 

Lawrence M. Aleamoni and llakonnen Yimer 
University of Illinois * 

A plethora of research studies has been conducted In the area of teacher 
effectiveness. Domas and Tiedeman (1950) have cited over 1000 studies In this 
area, with a variety of interpretations and assumptions concerning what a "good 
teacher" is. As an example, according to Sister Long (1957) "a teacher is 
effective when he does things or behaves in ways that stimulate the learning of 
understandings, skills, desirable attitudes and habits, and adequate personal 
adjustment. Changes must include all-* around pupil growth: intellectual, social, 
emotional, aesthetic, and spiritual" (p. 220). 

If we accept Guthrie's (1949) contention that teaching performance is best 
judged by students and colleagues, then we would expect a positive relationship 
to exist between the two judgements. Guthrie also asserted that student ratings 
represent more valid judgements than colleague ratings due to the fact that 
students spend more time in contact with the particular teacher than a faculty 
meniber does. It is important to note that colleague rating may be affected or 
influenced by acquaintanceship, student hearsay, the effect of the ratee's 
instruction on the rater's students, and inferences based on the academic records 
of the ratee. 

There are few studies that deal with the relationship between teaching 
effectiveness and publication (research productivity). For example, Guthrie 
(1949), and Voeks (1962) found that there was no significant association between 
research contributions and teaching effectiveness. More recently, Stallings and 
Slnghal (1969) reported a statistically significant but small correlation between 
publications and student evaluation. 

In determining effective teaching, therefore, researchers have used a variety 
of methods, most popular among which are peer (colleague) rating, administrator 



(supervisor) rating, student rating, self-rating and rating by number of publi«- 
cations* This study deals with teacher effectiveness with respect to the re* 
latlonship among student ratings, ratings by fellow teachers (colleagues), and 
research productivity* 
Method 

During the fall of 1969-70 a questionnaire (see Appendix A) was sent out 
to the University of Illinois faculty (Urbana-Champaign Campus) asking them to 
nominate or indicate three faculty members whom they felt deserve mention for 
good teaching* Individual faculty members are rated according to the frequency 
of nomination which ranged from 1 to 26. Therefore, this study does not include 
faculty members who were not nominated* Academic rank and sex of each nominee 
was also determined* Academic rank was defined as the academic rank held by the 
faculty member at the University of Illinois during 1969-70* The academic rank 
consists of teaching assistant, instructor, assistant professor, associate pro-* 
fessor, and full professor with scale values of 1, 2, 3, 4 and 5 respectively. 

Two student evaluation questionnaires which are used to collect student 
attitude data toward instructors anJ courses at the University of Illinois were 
used (see Appendix B) to obtain the student evaluation data for the nominated 
faculty members. The Illinois Course Evaluation Questionnaire (CEQ) is a SO item 
questionnaire that is used to elicit student opinions about a course. It is 
composed of six subscales: (a) General Course Attitude, (b) Method of Instruction, 
(c) Course Content, (d) Interest and Attention, (e) Instructor, and (f) Specific 
Items (Spencer and Aleamoni, 1970)* The student responds by indicating his agree- 
ment or disagreement on a 4-point scale where 1 represents least favored and 4 
best favored with respect to teaching effectiveness* The second student evalu- 
ation questionnaire used was The Advisor, a 34 item questionnaire also used in 
evaluating courses* It is divided into four sections or subscales: (a) Overall 
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Evaluation of Course, (b) Instructor, (c) Quiz or Discussion, and (d) Laboratory 
or Language Lab. The two major parts of The Advisor (Overall Evaluation of 
Course and Instructor) apply to all types of courses. The scale values for The 
Advisor are the same as the CEQ. 

The CEQj composed of 7 variables (the six subscales plus a Total for all 50 
items), and The Advisor, composed of 2 variables (Overall Evaluation of Course and 
InstructorX were collected for all nominated faculty members who had one or both 
student evaluations for the fall and spring aemester of 1969-70. Due to the 
scarcity of the number of faculty members who had CEQ ratings, the collection of 
CEQ ratings was extended as far back as 1966-67. For those individuals who had 
more than one CEQ or Advisor rating, the average was computed. 

In addition, publications of the faculty were collected from a publication 
entitled "Publications of the Faculty and Titles of Doctoral Dissertations" (1966, 
1967, 1968, 1969) put out yearly by the University of Illinois. Senior author- 
ship was not distinguished in the collection of the data. The listing gives the 
author, co-author(s) , title and bibliographic citation of all the books, arti- 
cles, technical reports and bulletins, book reviews and doctoral dissertations 
published by the university faculty. Publications in the present analysis con- 
sisted of the weighted and unweighted sum of frequency counts of five variables 
(books, books edited, articles, reviews, and bulletins and technical reports). 
The weighting scheme used on the five publication variables was that suggested by 
Stallings and Singhal (1969) and is presented in Table 1. Four hundred seventy- 
seven different faculty members were nominated at least once. Of these, CEQ data 
were available for 43, Advisor data for 474, and publications for 362. A corre- 
lational analysis was used along with multiple regression on the variables of 
interest. 
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Table 1 

Weights Assigned to Publications 



Publication 


Weight I 


Weight II 


Books 


15 


9 ' 


Books Edited 


9 


4 


Articles 


3 


3 


Book Reviews 


2 


2 


Bulletins and 
Technical Reports 


3 


3 



Results 

A description as well as the mean and standard deviation of the variables 
are presented in Table 2. The distribution of rating by other instructors 
(Varirble 3) was positively skewed, i.e., the majority of the faculty members 
had a nomination of 1, 2, 3 or 4. This was also indicated by. the low mean 
(Table 2). For the same group of individuals the average academic rank (Variable 1) 
seems to fall between assistant professor and associate professor. 

In Table 3 the data above the main diagonal represent the intercorrelations 
among the variables while the data below the main diagonal represent the corres- 
ponding sample sizes. 

The average of the intercorrelations of the CEQ variables (6, 7, 8, 9, 10 

and 11) excluding the Total CEQ variable (12) is about .85 and the average cor- 
relation of each CEQ subscale variable with the Total CEQ variable is .93. This 
high intercorrelation among the CEQ variables is an indication that variable 12 
(CEQ Total) should account for most of the variance that variables 6, 7, 8, 9, 
10 and 11 would account for separately. Hence, variable 12 may be used as a 
measure of teacher effectiveness without considering the rest of the CEQ variables. 
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Table 2 

Variable Description, Mean and Standard Deviation 



V i\ J\ XADl^ & O 


N 


MEAN 


S.D. 


1. Academic Rank 


477 


3.83 


1.17 


2. Sex 


477 


1.12 


.32 


3. Colleague Rating 


477 


2.98 


3.14 


AGvisor ouDscaj.es 








H« uverail cva±uacxon or i^ourse 




3 11 


.28 


5» Instructor 


47A 


3.22 


.39 


CEQ Subscales 








6* General Course Attitude 


43 


3.16 


.35 


7. Method of Instruction 


43 


2.84 


.42 


8. Course Content 


43 


2.97 


.24 


9» Interest and Attention 


43 


2.85 


.42 


10. Instructor 


43 


3.22 


.34 


11 « Specific Items 


43 


2.95 


.20 


12. Total 


43 


3.00 


.30 


13. Publications 


362 


7.89 


7.55 



The correlation between variable 4 (Advisor Overall Subscale) and variab).e 
5 (Advisor Instructor Subscale) was 0.84. Unlike the CEQ, the Advisor lacks an 
Advisor Total scale that combines variables 4 and 5. However, due to the high 
correlation, variable 5 may be used to represent variable 4 in the measurement 
of teaching effectiveness. 

Table 4 presents the intercorrelations of variables 1, 3, 5, 12 and 13. 
The correlation between variables 1 and 13 (Academic Rank and Pub.lication) is 
0.32 and is significant at p<.05. The correlation of variable 3 with academic 
rank. Advisor Instructor and CEQ Total is 0.20, 0.28, and 0.27, respectively, 
and all of them are significant at p<.05 level. In order to determine the con- 
tributions of Academic Rank Advisor, CEQ, and Publicatipn in predicting colleague 
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rating a multiple correlation and the weights for each predictor variable was 

computed. The multiple correlation was found to be .40 and the standardized 

and unstandardized regression equations are presented below: 

Zy « (-6722) CEQ + (.6667) Academic Rank + (•3217) Advisor 
- (.0112) Publications 

^ « 1.554 + (2.694) CEQ + (.689) Academic Rank + (-990) Advisor 
- (.001) Publications 

f\j 

Where Zy = predicted colleague rating (standardized) 

Y «= predicted colleague rating (unstandardized) 



Table 4 
Selected Intercorrelations 



VARIABLES 


1 


5 


12 


1^ 


3 


\. Academic Rank 

5. Advisor Instructor 

12. CEQ Total 

13. Publications 

3. Colleague Rating 


1.00 


.03 
1.00 


- .23 
,58* 
1.00 


.32* 

- ,02 

- .04 
1.00 


,20* 
.28* 
,27* 
.07 
1,00 



* Significant at .05 level. 



Discussion 

It is evident from the results presented that the academic rank of an 
instructor seems to be more highly related to publications thafl to student 
opinion^ sex or colleague rating. Colleague rating as well as student eval- 
uation failed to show a significant relation with publications, but colleague 
rating was significantly relat^ed to academic rank. One would conclude that an 
instructor's rating by colleagues is unre3.ated to the number of publication(s) 
that the instructor has produced and, therefore, that publication may not be an 
important factor in rating £U instructor's teaching. Guthrie (1949) and Voeks 
(1962) also arri\2d at the same conclusion in their study. However, the 
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Stallings and Singhal (1969) results seem to disagree with the above conclusion* 
It is important to note that the sample size ^^sed for the correlaJbion between 
CEQ Total and Publication 01=28) was about 1/4 of that of Stallings and Singhal 's 
report. But the sample size for the Advisor Instructor (N«360) is about 3 times 
that of Stallings and Singhal* s. As already indicated the CEQ and Advisor ratings 
are highly correlated, implying that the sample size differences between the 
present study and Stallings and Singhal* s may not be responsible for the different 
results. 

i<:\nother implication is that teachers and students differ in the basis of 
their rating since instructors seem to take into consideration academic rank of 
the instructor in their rating while this is not the case for students. However, 
this relationship is explainable in norms of reputation, as an instructor is at a 
university longer and is apt to be known to mote colleagues. On the other hand, 
students are rating the actual performance they observe and, hence, should not 
be affected by reputation. 
Stunmary 

Although the present study was aMe to provide data concerning the relation- 
ship of publications to instructional ratings and the instructor's academic rank 
to his instructional rating, there are other related issues that need to be in- 
vestigated. For example: (a) Would the relationship between an instructor's 
academic rank, publication, and student rating be the same if all instructors 
rated by students were used? (b) Do raters at a particular academic rank tend 
to rate higher, instructors at that same rank? (c) Does the reputation of an 
instructor affect student ratings? 
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Appendix A 



TO: Faculty Member Addressed 

FROM: Richard E. Spencer 

SUBJECT: Faculty Poll for Effective Teaching 

DATE: November 25» 1969 



In order to refine our measurement of effective teachings we ^'Ould 
like to compare student opinion of god3v^ teaching with faculty opinion. To 
this end we would like you to indicate below who you believe is a good teacher 
on this campus — defined as you see fit. Please nominate or indicate three 
people who you feel are, in fact, good teachers and return this form in the 
enclosed envelope. Please print the names and departments (or courses) of 
one to three faculty members at the University of Illinois Urbana campus whom 
you feel deserve mention for good teaching. 



1* 



Last Name, 



First 



Department 



Last Name, 



First 



Department 



3. 



Last Name, 



First 



Department 



1 ^ 



Appendix B 
The Advisor 

Illinois Course Evaluation Questionnaire - Form 66 
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Side Two 



THE ADVISOR 



is exces^i/e. 



C7 .au^ 

OMIT 1TE!^1S WHICH DO NOT APPLY TO THIS COURSE 
OVERALL EVALUATION OF COURSE T^ese »ie:>s e'aie orny !< irp L^-.i:[e se:: ■ n 
I :eaf-)ed :i vent deal m tniS course, 
J enjoyed the course, 

Tco Tuch ureievntil '^.ater-al was presented 
Exof^io^-itjons -riain'y tested trufo. 
Grading was fcjsed on clear standards. 
T'^e amount o* ..or^ required for th's course 
Ccntent of exar.tnat'ons v.as up.fatr. 
This cou'se -ns fcg?> object. ves. 

Out - c* • cJass '.vork i.e. Hon'e.'.o''k- .\3s rele/ant *:nd heipfu? 
Graatng 'n thts ccufse nas been unfa*r. 
T*-e course Tjterta! see'red /.orihw^^ite. 
AM that ts required for th»s course is mert.oruation. 
CoTtenl CI the course a3S good. 
T'ws couri>e .%"s a waste ot lime. 
Note taking m the lecture -.vas difficult. 
THrs course could be consider-'^b'y imorov-ed 

C;;ns{ jer>i»? 'j'' hufT^e.vcri- ".e. papers, p't'iects 'eadu»s^ he 
A 0-? B. 4-6 C. 7-9. 0. 10 or rore 
Vi'nat types exams were given'^ 

A. Objective or rnathecat-ca! pfobieir; solving cn > B. 
I NSTRUCTOR »'Tvse ite^^s 'elali only to the rr.a.n mstri.ctcr.j 
WfMO youf :nam instructor's naf^ie 

The presentations by the instructorts .vere excellent. 

The instructor seems to dislike teaching thJS course. 

The instructor was interesting. 

The personality of the instructor was a problem. 

1 would like to take another course taught by th»s mstmctoi. 

In general. I rank this instructor h-gher than others I have had. 

QUIZ OR DISCUSSION (These items relate only to the Qui/ cr Dsscussion sectton. 
Write your Quiz or Oisrussion instructof's nane <T A.» 

♦ ' This class helped to clarify texts and lectures. 

There \vas ample opporicnity to ask questions. 

The T.A. did not cover enough of the subject matter introduced in lectures. 
' * This T.A. IS an effective instructor. 

Questions were ansv/ered clearly and appropriately. 

This class was valuable to the understandi.ig of the course. 



SAMPLE MARKS 
USE 

PENCIL 
ONLY 

RESPO.NSE CODE 

'/APK 



an> tours 



C. 6ot^r A. .i^C B. 



LABORATORY OR LANGUAGE LAB (These items relate only to the Laboratory or La'iguage Lab.i 
Write your Laboratory mstructor's narr.e 

The lab assistant v/as very helpful. 
There were too many problems v/ith the lab equipment. 
Too much time v/as wasted m the lab. 



4 ~ 



j Lab v/ork was v/ell organized. 
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, . THE ADViSOR 

Side One 

f*icase jse thts side J th- for^- fO? y.;u' sop.^' ,o. menls or teacr»6' ^(f^:<^;^^.^s j ie.ica; 



COURSE CONTENT 



iNSTRUCTORS ^- :"e n.. e ; ;.;r . : ; e ^st:.:::r 

V. ra* are >cu: ?encra^ .o-^ -ens 3r. ;t ine r.sujti:" ? t*^ s :o.'se 



PAPERS AND HOMEWORK 

Cc^'-.eni on the value of t)oo^^s. no-':e.vDrK. and Daoers tr any - m tnis course. 



EXAMS 

Con-ent on the exars 'quizzes. p?act;ca!S' as to aifficufty. launess, eic. 



GENERAL 

:. nhat 'rpfovef^ei^is lo :h s c:u(se /;oula vou suggest' 



2 P''*3segive /our {hcbt;!itf J %/a'.jfori of Ihi: course .vith conr^ents. A^e ;G'. SdV$^eJ .•. !h .v'-^t 
D: y.->u consider i! 3 /aluaVe edbcaticnal experienced S'niply a neans rassmi <j reOuje^^ 
Please comment. 
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U.LIXOIS COI RSK K\ ALUATiON Ql'EftTIONN AT R K 



— FORM br> 



Ci Wi v> J 

' < r- ^ 



^ d 
> > 



or 

u o 

2 5.-; 

H a: < 
2 < u. CD 

w| o 
^ a 

4 CO to 

a: 



o 



a 'J q: ^ 



CO 

z 

O 

r- 

O 

ft; 
5 



J ' $ t r L i c ! \ se e " 0 ci to^ »nie'esteJ . - students :;erso s. 



It /-<js e.»sy to JO" *itie .tf^e. 

Tre ' ' ^tr .etc 3"* ^ "^i ^ t*'es.zo, k tr^f ' ,3^ ze t^ff»'Ct. 

N.'t - w-^ ^s. t\T j wOy t -K'lT rriS course. 



SAMPLE MARKS- 
USE 

PENCIL 
ONLY 



RLSPON'SE CODE 

A " " f • • 



H c " e r^ct^ s-ss > g'-; " ' & -c i r f 1 1 ! > ] uti iter s t n g the cci.rse. 

l'^ e oi^gh v^.aeit pjtt i ctp.-^tion ^of this type ot cj ,rsc. 
T ^o >rstf ^;tc a t"Qf.:j jr> kn.'/.l e c o f h < s stj bj ec t ''m 1 1 ef . 
TSe t'.''tiei't cf the course /.as pooc!. 
The co-^i so defeased generaf J^aoz/ledge. 



' -AP" ' 3 



The types of lest Jnestjorss used .vere good. 


COMPlETc SECTlCtsIS BELOu ACCORDING I 
TO VOUR INSTRUCTOR'S DIRECTIONS: 


Held ^ly attent oi th.-ooghOLt the coutse. 


The '^enarids of the sttdents v.ere not considered by the jnst;uctc/. 


OPTIONAL 
PART !I 
ITEf/.S 51-75 




OPT!0^iAL 1 
PART !fi j 

ITEMS 76.100 


Unmierestms: cojfse. 


It ••.<!$ a ve^y '.vortJv.%h le cojrse 



Sop'e thtngs A'ere not explained very v/e!l 



T^e .-.ij n v.-n;ch thts coL.'se v/as taugt^t resiilts in better student iearrung. 



The course ■'Mte' I al .vas too difficutt. 
Oi^e of niy poorest courses. 



iMatertal m the course was easy t o follo'»v. 

The fnstrucvo r seed ed to co nsider teachtng as a chore or routine actIv^^y. 
More outside reading is necessary. 



Course Tiatenal v/as poorly organized. 
Course was not very helpfuL 
It v/as quite interesting, 



I think that the course was taught quite well. 



I w ould prefer a different meth od of instruction. 
The pace of the course was too slow. 
At times I v/as confused. 



Excellent course content. 



The exarinnattons v/ere too difficult. 



Generally, the course v/as v/ell organized. 



Ideas ano conce pts were developed too rapidly. 
The content of the course v/as too elementary. 



Some days I was not very interested in this course. 
It v/as quite bormg. 

The instructor exhibited professional dignity and bea ting in the classroom. 
Another method of instruction should have bee n employed. 
Tl^e course was quite useful. 



I v/ould take another course that was taught this way. 
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